Updating DLCLive!-GUI to pyQt6 as well as DLC3/pytorch models #34
Conversation
…modern-python-and-pyqt6 Add Basler and GenTL camera backends for modular capture
…era-functionality
…camera-functionality Rework layout and camera handling controls
…' into artur/test_update
…r integration - Implemented `get_device_count` method in `GenTLCameraBackend` to retrieve the number of GenTL devices detected. - Added `max_devices` configuration option in `CameraSettings` to limit device probing. - Introduced `BoundingBoxSettings` for bounding box visualization, integrated into the main GUI. - Enhanced `DLCLiveProcessor` to accept a processor instance during configuration. - Updated GUI to support processor selection and auto-recording based on processor commands. - Refactored camera properties handling and removed deprecated advanced properties editor. - Improved error handling and logging for processor connections and recording states.
There was a problem hiding this comment.
Pull request overview
This PR modernizes the DeepLabCut-live-GUI by migrating from Tkinter to PyQt6 and adding support for DLC3/PyTorch models. The update includes hardware-accelerated video encoding (NVENC), a new processor selection system with on-the-fly changes, remote control capabilities via processor sockets, and user-definable bounding box visualization.
Key Changes:
- Complete UI rewrite from Tkinter to PyQt6
- Added support for PyTorch-based DLC models
- Implemented hardware-accelerated video recording with NVENC
- Added processor plugin system with remote control capabilities
- Introduced multiple camera backends (OpenCV, GenTL, Aravis, Basler)
Reviewed changes
Copilot reviewed 48 out of 49 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| setup.py | Updated version to 2.0, Python requirement to >=3.10, replaced dependencies for PyQt6 |
| pyproject.toml | Added modern Python packaging configuration with comprehensive test setup |
| docs/user_guide.md | New comprehensive user guide for camera setup, DLC configuration, and recording |
| docs/timestamp_format.md | Documents JSON timestamp format for frame synchronization |
| docs/features.md | Detailed feature documentation including camera backends and processor system |
| docs/camera_support.md | Camera backend comparison and installation guides |
| docs/aravis_backend.md | Aravis backend documentation for GenICam cameras |
| docs/README.md | Documentation index with navigation by use case and topic |
| dlclivegui/video_recorder.py | New video recording module using vidgear with hardware acceleration |
| dlclivegui/gui.py | Complete PyQt6-based GUI implementation replacing Tkinter |
| dlclivegui/dlc_processor.py | DLCLive integration with threading and performance profiling |
| dlclivegui/config.py | Configuration management with dataclasses |
| dlclivegui/cameras/opencv_backend.py | OpenCV camera backend implementation |
| dlclivegui/cameras/gentl_backend.py | GenTL/Harvesters camera backend |
| dlclivegui/cameras/factory.py | Camera backend factory and detection |
| dlclivegui/cameras/base.py | Abstract camera backend base class |
| dlclivegui/processors/processor_utils.py | Processor plugin loading utilities |
| dlclivegui/processors/dlc_processor_socket.py | Socket-based processor with remote control |
| dlclivegui/processors/PLUGIN_SYSTEM.md | Processor plugin system documentation |
Comments suppressed due to low confidence (1)
dlclivegui/processors/processor_utils.py:1
- The docstring 'Returns:' section contains what appears to be an absolute file path rather than a description of the return value. This should describe the returned dictionary structure instead.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
dlclivegui/gui.py
Outdated
| from dlclivegui.processors.processor_utils import instantiate_from_scan, scan_processor_folder | ||
| from dlclivegui.video_recorder import RecorderStats, VideoRecorder | ||
|
|
||
| os.environ["CUDA_VISIBLE_DEVICES"] = "1" |
There was a problem hiding this comment.
Hardcoded environment variable and absolute path should not be in production code. These appear to be development-specific settings that should be removed or made configurable through the configuration system.
| os.environ["CUDA_VISIBLE_DEVICES"] = "1" | |
| # If you need to restrict CUDA devices, set CUDA_VISIBLE_DEVICES externally or via configuration. |
dlclivegui/gui.py
Outdated
|
|
||
| logging.basicConfig(level=logging.INFO) | ||
|
|
||
| PATH2MODELS = "C:\\Users\\User\\Repos\\DeepLabCut-live-GUI\\dlc_training\\dlclive" |
There was a problem hiding this comment.
Hardcoded environment variable and absolute path should not be in production code. These appear to be development-specific settings that should be removed or made configurable through the configuration system.
| PATH2MODELS = "C:\\Users\\User\\Repos\\DeepLabCut-live-GUI\\dlc_training\\dlclive" | |
| # Use environment variable DLC_PATH2MODELS if set, otherwise default to a relative path | |
| PATH2MODELS = os.environ.get("DLC_PATH2MODELS", str(Path(__file__).parent.parent / "dlc_training" / "dlclive")) |
There was a problem hiding this comment.
@arturoptophys see copilot suggestions; please let me know if you would like assistance!
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
| return True | ||
|
|
||
| def stop(self) -> None: | ||
| """Request a graceful stop.""" |
deruyter92
left a comment
deruyter92
left a comment
There was a problem hiding this comment.
Amazing improvements, @arturoptophys!
I've tested it with python 3.10, 3.11 and 3.12. Only ran with pytorch models. Only a few specific comments/questions. Overall great improvement of the user experience and well-written code!
| def stop(self, wait: bool = False, *, preserve_pending: bool = False) -> None: | ||
| if not self.is_running(): | ||
| if not preserve_pending: | ||
| self._pending_settings = None | ||
| return | ||
| assert self._worker is not None | ||
| assert self._thread is not None | ||
| if not preserve_pending: | ||
| self._pending_settings = None | ||
| QMetaObject.invokeMethod( | ||
| self._worker, | ||
| "stop", | ||
| Qt.ConnectionType.QueuedConnection, | ||
| ) | ||
| self._worker.stop() | ||
| self._thread.quit() | ||
| if wait: | ||
| self._thread.wait() |
There was a problem hiding this comment.
Is there a reason why self._worker.stop() is called twice? Otherwise only keep the QueuedConnection.
| # Try grab first - this is non-blocking and helps detect connection issues faster | ||
| grabbed = self._capture.grab() | ||
| if not grabbed: | ||
| # Check if camera is still opened - if not, it's a serious error | ||
| if not self._capture.isOpened(): | ||
| raise RuntimeError("OpenCV camera connection lost") | ||
| # Otherwise treat as temporary frame read failure (timeout-like) | ||
| raise TimeoutError("Failed to grab frame from OpenCV camera (temporary)") |
| # Set FPS if specified | ||
| if self.settings.fps: | ||
| self._capture.set(cv2.CAP_PROP_FPS, float(self.settings.fps)) | ||
|
|
||
| # Set any additional properties from the properties dict | ||
| for prop, value in self.settings.properties.items(): | ||
| if prop in ("api", "resolution"): | ||
| continue | ||
| try: | ||
| prop_id = int(prop) | ||
| except (TypeError, ValueError): | ||
| continue | ||
| self._capture.set(prop_id, float(value)) | ||
|
|
||
| # Update actual FPS from camera | ||
| actual_fps = self._capture.get(cv2.CAP_PROP_FPS) | ||
| if actual_fps: | ||
| self.settings.fps = float(actual_fps) |
There was a problem hiding this comment.
Maybe useful to add logging here. E.g. if a property fails to set. Or when the actual_fps turns out to be different from self.settings.fps.
There was a problem hiding this comment.
added logging if parameters dont go through or are still set to unexpected values
| try: # pragma: no cover - optional dependency | ||
| from dlclive import DLCLive # type: ignore | ||
| except Exception: # pragma: no cover - handled gracefully | ||
| DLCLive = None # type: ignore[assignment] | ||
|
|
There was a problem hiding this comment.
I think it would be definitely useful to log a warning here that DLCLive is not imported.
Also: printing the exception telling why DLCLive was not imported helps users debug cases where they unsucessfully tried to install DLCLive (e.g. DLCLive is installed, but one of the dependencies missing, resulting in a failed import).
There was a problem hiding this comment.
added logging of the exception. would this be enough to pinpoint the issue?
| def _start_worker(self, init_frame: np.ndarray, init_timestamp: float) -> None: | ||
| if self._worker_thread is not None and self._worker_thread.is_alive(): | ||
| return | ||
|
|
||
| self._queue = queue.Queue(maxsize=1) |
There was a problem hiding this comment.
What is the reasoning behind maxsize=1?
There was a problem hiding this comment.
To only predict newest frames, as we want to use most up-to-date infos we dont need to care about older frames and just drop them
|
|
||
| ## DLCLive Configuration | ||
|
|
||
| ### Prerequisites | ||
|
|
||
| 1. Exported DLCLive model (see DLC documentation) | ||
| 2. DeepLabCut-live installed (`pip install deeplabcut-live`) | ||
| 3. Camera preview running | ||
|
|
There was a problem hiding this comment.
include link for step 1:
see DeepLabCut documentation
step 2:
I think deeplabcut-live (with pytorch) should be a dependency that is automatically installed, do you agree?
| 1. Click **Browse** next to "Model directory" | ||
| 2. Navigate to your exported DLCLive model folder | ||
| 3. Select the folder containing: | ||
| - `pose_cfg.yaml` | ||
| - Model weights (`.pb`, `.pth`, etc.) |
There was a problem hiding this comment.
I was thinking, would be nice to have an option here to directly download superanimal models from the modelzoo.. What do you think? If you want I can make a follow-up PR for this.
There was a problem hiding this comment.
Sounds great! Would need to run export models to dlclive format as well ! Also as many of superanimal models have detector we also need to fix this DeepLabCut/DeepLabCut-live#137
|
|
||
| dependencies = [ | ||
| "deeplabcut-live", | ||
| "PyQt6", | ||
| "numpy", | ||
| "opencv-python", | ||
| "vidgear[core]", | ||
| "matplotlib", | ||
| ] |
There was a problem hiding this comment.
There are many branches of deeplabcut-live and it has recently seen quite some updates. Since not everything is backward compatible, I think it will be important to pin a specific version here.
There was a problem hiding this comment.
https://github.com/DeepLabCut/DeepLabCut-live/tree/maxim/dlclive3 this is the branch of dlclive I worked with
|
|
||
| [project.optional-dependencies] | ||
| basler = ["pypylon"] | ||
| gentl = ["harvesters"] | ||
| all = ["pypylon", "harvesters"] |
There was a problem hiding this comment.
Wouldn't it be useful to take over the optional torch and tensorflow dependencies of deeplabcut-live?
(These packages are exclusively used in the deeplabcut-live backend, and the backend specifies which torch/tensorflow work well).
i.e. to summarize my suggestions for pyproject.toml:
- default dependency deeplabcut-live with pytorch
- optional dependency deeplabcut-live with tensorflow
- pin the required version of deeplabcut-live to branch dlclive3
dependencies = [
"deeplabcut-live[pytorch]",
"PyQt6",
"numpy",
"opencv-python",
"vidgear[core]",
"matplotlib",
]
[project.optional-dependencies]
tensorflow = ["deeplabcut-live[tf]"]
basler = ["pypylon"]
...
[tool.uv.sources]
deeplabcut-live = { git = "https://github.com/DeepLabCut/DeepLabCut-live.git", branch = "dlclive3" }
There was a problem hiding this comment.
Shall we empty this setup.py, since everything is in pyproject.toml? Just leave something like this for backward compatibility:
from setuptools import setup
# All configuration is now in pyproject.toml
# This file is kept for backward compatibility with tools that don't support pyproject.toml
setup()
|
|
||
| init_start = time.perf_counter() | ||
| options = { | ||
| "model_path": self._settings.model_path, | ||
| "model_type": self._settings.model_type, | ||
| "processor": self._processor, | ||
| "dynamic": list(self._settings.dynamic), | ||
| "resize": self._settings.resize, | ||
| "precision": self._settings.precision, | ||
| } | ||
| # Add device if specified in settings | ||
| if self._settings.device is not None: | ||
| options["device"] = self._settings.device | ||
| self._dlc = DLCLive(**options) |
There was a problem hiding this comment.
Note: options is missing a parameter single_animal. In the current DeepLabCut-live version 3 from maxim, this boolean parameter defaults to True. So even when a multi-animal model is running, DeepLabCut live is configured to treat it as a single-animal setup. The consequence is that for multiple pose outputs only the first one is selected.
We might choose to accept this behavior as convenient for now, while we think about the required adjustments for the multi-animal case. Meanwhile I think it would be good to include the argument single_animal: True here, to make it explicit here that you are launching DeepLabCut-live in single animal mode.
There was a problem hiding this comment.
i have added single_animal=True as explicit parameter into the configs. once we add the option to actually deal with multi-animal predictions we can expose this parameter to be configurable.
- Update configuration settings to include support for single-animal models in DLCProcessorSettings. - Improve user guide with a link to DLC documentation for model export.
…le_animal in DLCLiveProcessor settings
4 participants
This updates the GUI to pyQt6.
New features: